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DETAILED ACTION 
Response to 08/07/2007 Amendment 

1 . The amendment filed on 08/07/2007 has been entered. 

Response to Arguments 

2. Applicant's arguments filed 08/07/2007 have been fully considered but they are 
not persuasive for the following reasons. 

The arguments concerning the 35 USC 112 first paragraph rejection have been 
fully considered but they are not persuasive because the claimed "real object" is similar 
to the objects in real space that are measured by stylus 1060 in order to detemiine the 
3D coordinates of the constraining shape. See applicants specification at [0043], 
[0051], [0052], and [0079] (at line 5 note "constraining plane 2000 on the real object"). 
According to applicants specification points of the "real object" are received from stylus 
1060 and are not received by moving the "real object". Alternative means for stylus 
1060 are briefly mentioned in paragraphs [0028], [0041], [0050], [0068], and [0078], 
however, points of the "real object" would then be received from an alternative to stylus 
1060 and would not be received by moving the "real object". Thus, applicants use in the 
originally filed specification of "object" with regards to real space and applicants use in 
the originally filed specification of "real object" defines "real object" differently than the 
amended claim define "real object", therefore, the originally filed specification does not 
convey that applicant moves the "real object" to obtain the points forming the 
constraining shape. The previous objection to the amendment filed 1/30/2007 with 
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regards to the amendment to the specification and the 1 12 first paragraph rejection is 
maintained and reproduced below. 

The arguments concerning the 35 USC 102 first paragraph rejection have been 
fully considered but they are not persuasive because as discussed by applicant in 
response to the 35 USC 112 first paragraph rejection and discussed in applicants 
specification at paragraphs [0028], [0041], [0050], [0068], and [0078] alternative means 
for stylus 1060 for inputting the 3D coordinates of the constraining shape is possible, 
thus, the claimed "inputting means for inputting three-dimensional position information 
of a plurality of positions inputted by moving a real object in the real space by a user, 
the real object being capable of measuring the position and orientation in the real 
space" of claim 1; "an inputting step of inputting three dimensional position information 
of a plurality of positions inputted by moving a real object in the real space by a user, 
the real object being capable of measuring the position and orientation in the real 
space" of claim 5; "obtaining three-dimensional position information of a plurality of 
positions designated by an operating unit moved by a user in the real space, the 
operating unit being capable of measuring the position and orientation" of claim 10; and 
"an inputting unit configured to input three-dimensional position information of a 
plurality of positions inputted by moving a real object in the real space by a user, the 
real object being capable of measuring the position and orientation in the real space" of 
claim 16 may be means other than stylus 1060 which alternative means is taught by 6 
DOF tracker device of Kitamura. Applicant amended the setting means of claim 1, the 
setting step of claim 5, the obtaining a constraining step of claim 10, and the setting unit 
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of claim 16 to claim "constraining shape by using a sliape generated based on the 
inputted (or obtained) three-dimensional information". The 6 DOF tracker device of 
Kitamura suggests using the 6 DOF tracl^er device to input 3D coordinates of the 
constraining shape since a means to input the constraining shape is inherently present 
and because the real object with inputted constrained properties is moved by the user 
using the 6 DOF tracker device which will allow for less input devices being used in 
Kitamura's system. Additionally section 2 at lines 7-14 states "To bring an object that 
already exist in the real world into a computer-generated virtual world, it is necessary to 
construct accurate shape representation of the real object in a computer system. A 
traditional method for this is to use conventional modeling software after precisely 
measuring the size or length of the real object by hand." which suggests using a 
computer input device to measure the real object in order to have accurate shape 
representation of the real object's constraining shape. The previous 102 rejection is 
therefore converted into a 103 rejection to address the amendments to the setting 
means of claim 1 , the setting step of claim 5, the obtaining a constraining step of claim 
10, and the setting unit of claim 16. 



Response to 01/30/2007 Amendment 

3. As discussed in the previous office action in paragraph 2 mailed on 05/07/2007, 
the amendment filed 1/30/2007 is objected to under 35 U.S.C. 132(a) because it 
introduces new matter into the disclosure. 35 U.S.C. 132(a) states that no amendment 
shall introduce new matter into the disclosure of the invention. The added material 
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which is not supported by the original disclosure is as follows: the changes made to 
paragraph [0006] enters new matter because the "object" is not clearly limited to the 
stylus discussed at paragraph [0028]. Applicant is required to cancel the new matter in 
the reply to this Office Action. 



Claim Rejections - 35 USC §112 

4. The following is a quotation of the first paragraph of 35 U.S.C. 112: 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the 
art to which it pertains, or with which it is most nearly connected, to make and use the same and shall 
set forth the best mode contemplated by the inventor of carrying out his invention. 

5. Claims 1-3. 5-9, and 16-18 are rejected under 35 U.S.C. 112, first paragraph, as 
failing to comply with the written description requirement. The claim(s) contains subject 
matter which was not described in the specification in such a way as to reasonably 
convey to one skilled in the relevant art that the inventor(s), at the time the application 
was filed, had possession of the claimed invention. The amendment to independent 
claims 1, 5, and 16 "input three-dimensional position information of a plurality of 
positions inputted by moving a real object in the real space by a user" is not conveyed 
by the originally field disclosure because an object is different than the stylus discussed 
at paragraph [0028] that would be moved by the user. Currently applicant is claiming to 
move any real object while the application only supports moving stylus 1060. Figures 2, 
10, and 12-14 shows stylus 1060 but do not convey that any real object may be moved 
to input three-dimensional position information. 
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Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

7. The factual inquiries set forth in Graham v- John Deere Co., 383 U.S. 1 , 148 
USPQ 459 (1966), that are applied for establishing a background for determining 
obviousness under 35 U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art, 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating 
obviousness or nonobviousness. 



8. This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 
prior art under 35 U.S.C. 103(a). 
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9. Claims 1-3 and 5-18 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Yoshifumi Kitamura and Fumio Kishino, Consolidated Manipulation of Virtual and 
Real Objects, September 1997, Proceedings of the ACM symposium on Virtual reality 
software and technology, pages 133-138. Kitamura teaches an augmented reality 
system that uses object constraints to control the visual interaction between the virtual 
objects and the real objects. 

A detailed analysis of the claims follows. 

Claim 10: 

Kitamura teaches an information processing method for changing the position 
and orientation of a virtual object in mixed reality space obtained by combining a real 
space and a virtual space, said method comprising the steps of: 

obtaining three dimensional position information of a plurality of positions 
designated by an operating unit moved by a user in the real space {In view of page 135 
first paragraph under tiie lieading of 5 Consolidated Manipulation Environment of 
Kitamura the same 6 DOF tracker device is used to control the position of a virtual world 
object(s) and to control the position of a constraining real world object(s) which is similar 
to applicants system where stylus 1060 is used to control the virtual world object(s) and 
to control the location of the constraining real world object(s), see applicants paragraphs 
[00281 [00301 and [0041]. The 6 DOF has a stylus the user moves which meets the 
"operating unit moved by a user in the real space" claim limitation.), the operating unit 
being capable of measuring the position and orientation in the real space {The 6 DOF 
measures position and orientation in real space of the sylus on the 6 DOF.)] 
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determining an input of a constraining shape or an operation of the virtual object 
{The system determines if a stiape having constraints is being input for the real object 
such as a toy block (section 4. 1) or surface (section 5.4) or such as moving the real 
object which has a constraining shape and the system detenvines if the user is moving 
the virtual object.)] 

Kitamura does not fully teach: obtaining a constraining shape by using a shape 
generated based on the obtained three-dimensional position information in the case of 
the input of the constraining shape {The first paragraph in section 5 on page 135 
describes the user using a 6 DOF tracker device to manipulate the virtual world objects 
and to position the constraining real world objects. The shape of the real world object(s) 
are used to constrain the movement of the virtual world object(s) by giving the real world 
object(s) a shape that the virtual world object(s) interacts with in a constrained manner A 
step of inputting the shape of the real world objects constraining shape is inherently 
present). Kitamura's use of the 6 DOF tracker device to control the position of the real 
world object suggests using the same 6 DOF tracker device to input the constraining 
shape of the real world object since this would require less input devices for the user to 
use and learn how to use. Additionally section 2 at lines 7-14 states "To bring an object 
that already exist in the real world into a computer-generated virtual world, it is 
necessary to construct accurate shape representation of the real object in a computer 
system. A traditional method for this is to use conventional modeling software after 
precisely measuring the size or length of the real object by hand." which suggests using 
a computer input device to measure the real object in order to have accurate shape 
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representation of the real objects constraining shape. Thus, it would have been 
obvious to one of ordinary skill in the art at the time of applicants invention to input with 
the 6 DOF tracker device the 3D coordinates of the constraining shape because the toy 
block (section 4.1) or surface (section 5.4) need to have their respective constraining 
coordinates input in order for the virtual object to properly interact with the real objects 
since this will require less input devices for the user to use and learn how to use and 
because section 2 at lines 7-14 suggests using a computer input device to measure the 
real object in order to have accurate shape representation of the real object's 
constraining shape. KSR International Co. v. Teleflex Inc., 82 USPQ2d 1385 (U.S. 
2007), U.S. Supreme Court No. 04-1350 Decided April 30, 2007, 127 SCt 1727, 167 
LEd2d 705.; 

Kitamura further teaches changing the position and orientation of the virtual 
object according to instructions from the user, based on the obtained constraining 
shape as constraint condition in case of the operation of the virtual object {The first 
paragraph m section 5 on page 135 describes the user using a 6 DOF tracker device to 
manipulate the virtual world object(s). The shape of the real world object(s) are used to 
constrain the movement of the virtual wortd object(s) by giving the real wortd object(s) a 
shape that the virtual world object(s) interacts with in a constrained manner Sections 
5. 1 to 5.4 discusses manipulation of the virtual wortd object based upon constraint 
conditions based on the shape of the real wortd object in response to the user using the 
6 DOF tracker device. The constraining shape generated from the three-dimensional 
position information constrains the interaction of the virtual wortd object with the real 
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world object, see sections 2, 4 to 5.4. The introduction on page 133 second full paragraph 
discusses augmented reality which synthesizes a virtual object with a real object. Sectbn 5. 1 
discusses after movement is detected by the 6D0F manipulator the virtual object is moved 
according to the constraints.); and 

combining an image of the virtual object generated according to the changed 
position and orientation, and the real image {The introduction on page 133 second hill 
paragraph discusses augmented reality which synthesizes a virtual world object with a real 
world object. Section 5. 1 discusses after movement is detected by the 6D0F manipulator the 
virtual world object is moved according to the constraints of the real world object. Sections 
2, 4, and 5 discusses mixing the virtual and real world images of virtual and real world 
objects.). 

Claim 11: 

Kitamura teaches an information processing method according to Claim 10, 
further comprising the step of combining a virtual image indicating the constraining 
shape with the real image {The introduction on page 133 second fiill paragraph discusses 
augmented reality which synthesizes a virtual object with a real object Section 5. 1 discusses 
after movement is detected by the 6D0F manipulator the virtual object is moved according to 
tiie constraints. Sections 2, 4, and 5 discusses mixing the virtual and real world images 
of virtual and real world objects.). 

Claim 12: 

Kitamura teaches an information processing method according to Claim 10, 
wherein the constraining shape is a plane. {Oi page 136 in the text €dx>ve figure 2 cBscusses 
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deteimining a plane and using the plane to constmh movement of the virtual objed is cBscussed with 
nBgan[ktofiguie2.). 
Claim 13: 

Kitamura teaches an information processing method according to Claim 10, 
wherein said changing step changing the position and orientation of the virtual object is 
carried out by changing the position and orientation of the operating unit {The 6 DOF 
tracker device is an operating unit. The user using the 6 DOF tracker device 
manipulates the virtual objects by changing the position and orientation of the 6 DOF 
tracker device.). 

Claim 14: 

Kitamura teaches a computer program product comprising a computer readable 
medium storing computer program code for performing the information processing 
method according to Claim 10 wherein the information processing method is executed 
by a computer device (This article is directed to computers that generate the augmented 
reality scene since it was published by ACM for a symposium on virtual reality software 
and technology and since at page 133 in the last sentence in the second paragraph of 
section 1 "software/hardware" is discussed. Software causing a computer to perform 
Kitamura' s augmented reality is inherently stored in a computer readable recording 
medium.). 
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Claim 15: 

Kitamura teaches a computer-readable recording medium, storing the computer 
program according to Claim 14 {Software causing a computer to perform Kitamura's 
augmented reality is inlierently stored in a computer readable recording medium.). 

Claim 16: 

The functions of Kitamura corresponds to the claimed units because the software 
causes the computer to become a unit that performs a process. As seen below 
Kitamura performs the claimed processes. 

Kitamura teaches an information processing device for aiding control operations 
relating to controlling the position and orientation of a virtual object, said device 
comprising: 

an image capturing unit configured to capture a real image in real space {The 
measurements by the user in three dimensional real space, the device for obtaining the 
3'D shape by using a range image, and the device for obtaining the 3-0 shape by using 
multiple cameras captures a real image in real space. The three paragraphs found in 
section 2 on page 134 of Kitamura teaches the user measuring the real objects, a device 
measuring the real objects with range in response to the user, or a device measuring the 
real objects with multiple cameras range in response to the user The claim does not limit 
the fonv of the real image, thus, Kitamura teaches this limitation. Note applicants paragraph 
[0024] does not restrict the image capturing unit to a particular type) ; 

a virtual image generation unit configured to generate a virtual image of a virtual 
object according to the position and orientation of said image capturing unit {Kitamura 
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discusses in section 2 in the first paragraph using conventional modeling software after 
precisely measuring the size or length of the real object by hand or by the devices which 
is an generation unit capable of generating three-dimensional positional information. 
Sections 2, 4 and 5 discuss generating a virtual image(s) of a virtual object(s).) ; 

a superimposing unit configured to superimpose the generated virtual image with 
the captured real image {Sections 2, 4, and 5 discusses mixing the virtual and real world 
images of virtual and real world objects.); 

a determination unit configured to determine an input of a constraining shape or 
an operation of the virtual object {The system determines if a shape having constraints 
is being input for the real object such as a toy block (section 4. 1) or surface (section 5 A) 
or such as moving the real object which has a constraining shape and the system 
determines if the user is moving the virtual object); 

an inputting unit configured to input three-dimensional position information of a 
plurality of positions inputted by a moving a real object in the real space by a user {In 
view of page 135 first paragraph under the heading of 5 Consolidated Manipulation 
Environment ofKitamura the same 6 DOF tracker device is used to control the position 
of a virtual worid object(s) and to control the position of a constraining real wortd 
object(s) which is similar to applicants system where stylus 1060 is used to control the 
virtual wortd object(s) and to control the location of the constraining real wortd object(s), 
see applicants paragraphs [0028], [0030], and [0041], The 6 DOF has a stylus the user 
moves which meets the "moving a real object limitation in the real space by a user"' 
claim limitation.), the real object being capable of measuring the position and orientation 
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in the real space {The 6 DOF measures position and orientation in real space of the 
sylus on the 6 DOF.); 

Kitamura does not fully teach: a setting unit configured to set a constraining 
shape by using a shape generated based on the inputting three-dimensional position 
information in the case of the input of the constraining shape {The first paragraph in 
section 5 on page 135 describes the user using a 6 DOF tracl<er device to manipulate 
the virtual world objects and to position the constraining real worid objects. The shape of 
the real world object(s) are used to constrain the movement of the virtual world object(s) 
by giving the real worid object(s) a shape that the virtual worid object(s) interacts with in 
a constrained manner A step of inputting the shape of the real worid objects 
constraining shape is inherently present). Kitamura's use of the 6 DOF tracker device 
to control the position of the real world object suggests using the same 6 DOF tracker 
device to input the constraining shape of the real world object since this would require 
less input devices for the user to use and learn how to use. Additionally section 2 at 
lines 7-14 states "To bring an object that already exist in the real world into a computer- 
generated virtual world, it is necessary to construct accurate shape representation of 
the real object in a computer system. A traditional method for this is to use conventional 
modeling software after precisely measuring the size or length of the real object by 
hand." which suggests using a computer input device to measure the real object in 
order to have accurate shape representation of the real object's constraining shape. 
Thus, it would have been obvious to one of ordinary skill in the art at the time of 
applicants invention to input with the 6 DOF tracker device the 3D coordinates of the 
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constraining shape because tlie toy block (section 4.1) or surface (section 5.4) need to 
have their respective constraining coordinates input in order for the virtual object to 
properly interact with the real objects since this will require less input devices for the 
user to use and learn how to use and because section 2 at lines 7-14 suggests using a 
computer input device to measure the real object in order to have accurate shape 
representation of the real object's constraining shape. KSR International Co. v. Teleflex 
Inc., 82 USPQ2d 1385 (U.S. 2007), U.S. Supreme Court No. 04-1350 Decided April 30, 
2007, 127 set 1727, 167 LEd2d 705.; and 

Kitamura further teaches an operating unit configured to control the position and 
orientation of the virtual object based on the constraining shape in accordance with the 
a user's instruction in case of the operation of the virtual object.( Sections 5,1 to 5A 
discusses manipulation oftlie virtual world object based upon constraint conditions 
based on the shape of the real world object in response to the user using the 6 DOF 
tracker device. The constraining shape generated from the three-dimensional position 
information constrains the interaction of the virtual world object with the real world object, 
see sections 2, 4 to 5.4, The introduction on page 133 second full paragraph discusses 
augmented reality which synthesizes a virtual object with a real object Section 5. 1 discusses 
after movement is detected by the 6D0F manipulator the virtual object is moved according to 
the constraints.). 

Claim 17: 

Kitamura teaches an information processing device according to Claim 16, 
wherein the constraining shape is defined by polygons {Figure 2's Cubes are formed 
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with polygons.) and the apexes of the polygons are at positions inputted by the user {As 
discussed in the a setting unit above it would have been obvious to use the 6 DOF 
tracker device to input the constraining shape of the real world object which would input 
the apexes of the constraining shape of the polygon,) or the constraining shape is a 
plane passing through the positions inputted by the user {A real sensed surface is a 
planar real world object whose position is inputted by the user in the setting unit. As 
discussed in the a setting unit above it would have been obvious to use the 6 DOF 
tracker device to input the constraining shape of the real world object which would input 
plane for a surface.). 
Claim 18: 

Kitamura teaches an information processing device according to Claim 16, 
wherein said operating unit performs at least one of the following operations in 
performing an operation controlling the position and orientation of the virtual object: 

a translation operation for causing translational movement of the virtual object 
based on the constraining shape {Page 137 column 1 lines 1-3 teaches translation 
when the virtual object collides with the real surface.)] or 

a rotation operation for rotating the virtual object on an axis which is a normal 
vector at a plane where the constraining shape and the virtual object come into contact 
{Page 137 column 1 lines 1-3 teaches rotation when the virtual object collides with the 
real surface.) {Page 137 column 1 lines 1-3 also teaches translation and rotation when 
the virtual object collides with the real surface). 
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Claims 1-3: 

Means plus function claims 1-3 correspond to device claims 16-18 and the 
means of Kitamura, software and computer, are equivalent to applicant's means of 
software and computer. 

Claims 5-7: 

Step for claims 5-7 correspond to device claims 16-18 and the steps of Kitamura, 
software and computer, are equivalent to applicant's steps performed by software and 
computer. 

Claims 8 and 9: 

Claims 8 and 9 mirror claims 14 and 15 addressed above and they are rejected 
for the same reasons given above for claims 14 and 15. 

10. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
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the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

1 1 . Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jeffery A Brier whose telephone number is (571) 272- 
7656. The examiner can normally be reached on M-F from 7:30 to 4:00. If attempts to 
reach the examiner by telephone are unsuccessful, the examiner's supervisor, Michael 
Razavi, can be reached at (571) 272-7664. The fax phone Number for the organization 
where this application or proceeding is assigned is 571-273-8300. 

Infomriation regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 

/Jeffery A. Brier/ 

Primary Examiner, Division 2628 



